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ABSTRACT 

This document presents the second year evaluation of 
the CAI/Cooperat i ve Learning Project* The purpose of this project is 
to develop a national model for integrating computer-assisted 
instruction (CAl) through an integrated learning system. The project 
is a collaborative effort by two Pennsylvania school districts, 
Hatboro-Horsham and Pittston Area, and Research for Better Schools 
(RBS) . This report begins with an 'ntroduction that provides 
background information on the project and the evaluation questions 
addressed by the study. Next, the evaluation design and procedures 
are described including design, student sample, instrumentation, and 
data collection and analysis. The findings of the evaluation 
questions for year two are presented in two sub-sect ions : program 
implementation and program outcomes. The program implementation 
sub-section presents data from both the Hatboro-Horsham and Pittson 
Area School Districts while the program outcome section includes data 
only from the Pittson Area. Finally, conclusions drawn from this 
evaluation and implications for year three are presented and 
examined. Included in the appendix are the surveys used in the 
evaluation as well as the mean survey responses for rated items. 
(JLB) 
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Introduction 

This report represents a second-year evaluation of the "CAI/Cooperative 
Learning Project." The three-year project is a collaborative effort by two 
Pennsylvania school districts, the Pittston Area School District and the 
Hatboro-Horsham School District, and Research for Better Schools (RBS) . It 
is federally supported by an Innovation in Education Program Grant. 

The introductory section of the report provides some background 
information on the project and the evaluation questions addressed by the 
study; later sections of the report describe the evaluation design and 
procedures, preliminary findings, and some conclusions and implications 
based on these findings. 
Background 

Both the Pitt:tton and Hatboro-Horsham School Districts had previously 
been involved in successful efforts to implement computer-assisted 
instructional technology. In order to further adopt this technology, in 
combination with innovative educational practices, the tvo districts jointly 
proposed this collaborative project to the Fund for Innovation in Education. 
Specifically, the project proposed to integrate advanced integrated learning 
system (ILS) technology with cooperative teaching in the classroom and 
collaborative learning in the home. It was anticipated that the project 
could potentially serve as a model for effective computer-assisted 
instruction that could be nationally validated, disseminated, and adopted by 
school districts across the nation. To realize this potential, the project 
needed a sound evaluation plan capable of demonstrating the full extent of 
its effectiveness . Because of its history in evaluating computer-assisted 
instructional programs, RBS was invited to coilabcrate with the two 
districts in the project, serving as a third-party evaluator. 



The project was to include the acquisition of computer hardware and 
software, initial and on-going teacher training t achievement and attitudinal 
data collection and analysis, and dissemination. For their software, the 
two school districts selected a computer-based learning system marketed by 
Jostens Learning Corporation. 
Evaluation Questions 

The purpose of the CAI/Cooperative Learning Project is to develop a 

replicable, independently validated, national model for integrating 

computer-assisted instruction through an integrated learning system that 

includes inquiry-based, hyper-media learning and cooperative learning 

techniques. An original feature of the project is the inclusion of 

cooperative education, creating a collaborative learning environment both 

within the classroom and at home. Based on the project goals, as specified 

in the project proposal, three outcome evaluation questions were formed to 

focus the evaluation study. They were: 

1. To what extent does the program enhance mathematics and language 
arts achievement for the participating students as measured by an 
increase in standardized test scores7 

2c To what extent does the program enhance positive student and parent 
attitudes toward learning , the integrated learning system, and 
education in general? 

3. To what extent does the program enhance positive teacher and 

administrator attitudes toward the integrated learning system and 
cooperative learning? 

The evaluation of the project's first year was primarily descriptive 

(Bi>yer, 1991) as the project was not initiated until the second half of the 

school year and, during this time, was not fully operational. It was hoped 

that an implementation focus for this first year would help to ensure that 

prerequisite conditions for proper program implementation were met. 

Throughout year two, RBS closely monitored implementation and program 
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operation in the two districts, and found implementation concerns to still 
be an issue, and paramount for the Hatboro-Horsham School District. This 
district experienced considerable hardware and software problems throughout 
the school year and, as a result of these unexpected and frustrating 
difficulties, the project reached only a minimal level of operation. Thus, 
the evaluation design, procedures, and outcomes described in this report 
apply mainly to the Pittston Area School District. In order to include a 
description of these critical implementation issues, the process evaluation 
question included in the first year report is included in this second year 
report as well. This fourth evaluation question is as follows: 
4. To what extent was the project implemented as planned? 
A discussion of the findings related to this evaluation question will set 
the stage for interpreting the outcome evaluation findings. 

Evaluation Design and Procedures 
This section describes the evaluation design, student sample, 
instrumentation, and data collection and analysis undertaken by RBS in 
connection with its second year evaluation of the CAI/Cooperative Learning 
Project implemented in two middle schools, one in the Pittston Area School 
District and one in the Hatboro-Horsham School District. 
D esign 

The approach used to address the evaluation questions was to include 
quantitative data collection of curriculum-embedded tests and standardized 
achievement tests. In addition, implementation and attitude measures were 
to be obtained through survey data collected at the end of the school year 
and through observations and informal interviews made during periodic visits 
to each of the sites. A pretest-posttest control group design was to be 
used (in years two and three) to enhance the validity of the findings, 



although there were differences in the nature of the control or comparison 
group at each school. For the Hatboro-Horsham School District, the control 
group was to be constituted from students within the same school who had no 
contact with computer-assisted instruction. For the Pittston Area School 
District, the control group was to consist of students within the school who 
had minimal contact with computer-assisted instruction. 

A number of changes were made in the evaluation design originally 
proposed. Major changes and their rationales are listed below. In terms of 
the proposed evaluation plan for collecting outcome data: 

# The TELLS data were not collected and analyzed as the state 
discontinued this testing program. 

# Outcome data from the Hatboro-Horsham School District were not 
collected and analyzed as this district experienced extensive 
implementation difficulties. 

O Parent attitude data from the Pittston Area School District were not 
included in the report (with the exception of home use survey data) 
as the district did not administer the parent survey. 

# Survey data were only collected at the end of the school year (for 
second year participants) as the previous end-of-year survey served 
as the pretest. 

In terms of the proposed evaluation plan fcr collecting process data: 

# Cooperative learning data were not collected as this portion of the 
program was not implemented by either district. 

# Time-on-task data were not collected and analyzed as preliminary 
observations indicated very high student engagement rates and very 
little variability . 

Further discussion of some of these issues can be found in the text. 
Student Sample 

Different strategies were used at each site to select program and 
control groups. In the Hatboro-Horsham middle school each grade is divided 
into two heterogeneous teams, a red team and a black team. For year one, 
the program group was selected from only the red team and consisted of the 



25 lowest-achieving students from each of the sixth, seventh, and eighth 
grades. All of these students had standardized achievement scores below 
grade level in mathematics and/or reading. The control group consisted of 
25 comparable stude^.s from each of the black team's three grades (these 
students did not have computers in their classes). For year two, the goal 
was to have the seventh and eighth grade program and control groups remain 
intact; the new sixth grade groups were to be assigned using the year one 
strategy. 

For year one, the Fittston middle school was divided into two groups 
based on standardized achievement test scores, an at-risk or program group 
(i.e., scoring below the grade level median) and a not at-risk group. All 
students in the at-risk group were then randomly assigned to one of two 
groups, a "low use" group which was to receive 15 minutes of computer time 
per week, or a "high use" group which was to receive 60 minutes of computer 
time per week (in 15 minute segments). There were approximately 50 students 
in each group at each grade level. To balance the low and high use groups, 
students assigned to one group for mathematics were to be assigned to the 
other group for reading. It should be noted that although the not at-risk 
group was also divided into low and high use groups, their data are not 
included in this report as the initial focus of the project was to be on 
low-achieving students. Again, for year two, the seventh and eighth grade 
groups were to remain intact, to the extent possible, and the incoming sixth 
grade groups were to be formed using the year one strategy. Alio, in each 
school, program students (five at a time) were to be on a rotating schedule 
to bring a computer home for a six week period. 



Instrumentation 

A number of evaluation instruments were developed by RBS, and approved 
by the districts, for use in the three-year study. They included the 
following : 

• Administrator Attitude Survey 

• Teacher Attitude Survey 

• Student Attitude Survey • 

• Parent Attitude Survey 

• Computer Home Use Survey. 

The administrator and teacher attitude forms contain questions which 
address concerns regarding the new computer leaiaing system, the advantages 
and disadvantages of the system, the adequacy of training and follow-up 
technical assistance, the implementation of the system, the adequacy of the 
computer curriculum, and the effectiveness of the computer learning system. 
The student attitude form addresses attitude toward using the computer in 
school, and the advantages and disadvantages of the computer learning 
system; and the parent attitude form and the home use form ask about parent 
involvement in the program, their child's attitude toward participation, and 
the advantages and disadvantages of the computer learning system. The home 
use survey is developed for parents of those students, who on a rotating 
basis take a computer home to increase their time on the system and their 
parents' involvement in collaborative learning. All surveys were also 
administered during year one. 
Data Co llection and Analys ij[ 

Process data were collected from both school districts participating 
in the project. These data were collected primarily through frequent 
informal interviews, telephone conversations, and on-site observations. 
Visits were made to the school's computer lab approximately every six weeks, 
during which time the computer coordinator or his assistant was informally 
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interviewed along with at least one reading and one mathematics teacher at 
each grade level. During classroom visits, students were observed working 
on computers whenever possible. Also, RBS • time-on-task measure was not 
used to collect data, as initial observations of students working on the 
computer, along with teacher feedback, indicated that students were highly 
engaged while working through their computer assignments and there was 
little variability in engagement rates. 

The Pittston Area School District also submitted outcome data for the 
project which consisted of completed attitude surveys and student 
achievement data (Stanford Achievement Test (SAT) basic skills data, Jostens 
student achievement reports). Attitude surveys were to be administered at 
the end of the school year, as year one data were to serve as a pretest. In 
cases where pretest data were not available (e.g., for the new sixth grade 
students), surveys were administered two times, at the beginning and end of 
the school year. Although some second year outcome data were collected from 
the Hatboro-Horsham School District, they are not included in this 
evaluation report. As previously mentioned, this district experienced 
extensive implementation difficulties, documentation of which is provided in 
the findings section of the report under the process or implementation 
evaluation question. 

Analyses were planned for both attitude and achievement data. 
Responses to items on the student, teacher, administrator, and home surveys 
were analyzed separately for each evaluation instrument. Also, only the 
student survey data were compared to the year one findings, as these surveys 
were both administered at the end of the school year. For the other 
surveys, comparisons were not made with the year one data as the 
pretest-posttest interval was too short (i.e., spring 1991 to fall 1991). 
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The SAT data were used to examine student achievement in reading and 
mathematics. In order to assess overall achievement gains, analyses were 
keyed to the students in sixth and seventh grade during year one. These 
analyses were carried out in terms of normal curve equivalents (NCEs) in 
order to look at students* achievement growth; an NCE change of zero 
indicates that students' achievement growth rate equals the rate of the 
representative national sample. In terms of the curriculum-embedded basic 
skills data, both initial placement and final lesson completed were made 
available as well as the amount of time students spent working on both 
lessons and unit tests. These data were analyzed to address project 
implementation and to set the stage for the discussion of outcomes by 
determining the number of lessons students completed during the year and the 
corresponding amount of time spent on the computer learning system. 

Findings 

The findings presented in this section of the report relate to the 
evaluation questions for year two. They are discussed below in two separate 
sub-sections, program implementation and program outcomes. The program 
implementation sub-section presents data from both the Hatboro-Horsham and 
Pittston Area School districts, the program zj' ~mes sub-section presents 
data from only the Pittston Area School District. 
Program Implementation 

Specific feedback on implementation of the project was obtained from 
informal interviews with teachers and computer coordinators, on-site 
observations, and telephone contacts. These data are discussed below under 
the implementation evaluation question. 
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To what extent was the project implemented as planned? 

In the fall of the 1991-92 school year, each of the two sites ordered 
and received additional hardware and software to continue their 
implementation of the project during its second year; program and control 
groups were updated; and schedules were developed for data collection and 
student computer use. Following these initial preparations, the two school 
districts attempted to implement the project, as described below. 

The Hatboro-Horsham School District experienced a number of problems 
with its implementation which were related to the school's new building, and 
to the functioning of both the computer hardware and software. In the 
summer of 1991, the Hatboro-Horsham middle school moved a short distance to 
a new location which previously served as the district's high school. In 
late September, federal funds had not yet been released to carry out the 
cabling necessary for the networked computer system to operate. The 
district then used its own funds, and the cabling was completed in 
mid-October. How'ever, upon completion, the new cabling did not work. It 
then took one month to identify the problem and to replace a defective 
cable. By mid-November, when the cabling became operational, project 
teachers began implementing the Jos tens matheir.acics curriculum as planned. 

Teachers and students experienced ongoing difficultly with the Jostens 
software (this district hdd purchased and installed the latest update of the 
sof tware--version 2.9). The major problems were that, while working on the 
computer learning system, students were frequently being presented with 
lessons which they had previously completed, and their progress was 
frequently not being stored in the system. These errors were reflected in 
student achievement reports; when data on lessons completed are not stored, 
all other reporting data (e.g., average scores, times, dates) are obviously 
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inaccurate. Although Jostens worked with the district throughout the school 
year to correct these problems, the problems continued to frustrate 
students, teachers, and district staff, to interfere with normal classroom 
operations, and to severely reduce enthusiasm for the computer learning 
system. 

One problem- solving strategy to address the difficulties experienced 
with the Jostens software was to "rebuild" the file server. Although this 
appeared to be successfully accomplished in March, the outcome was 
unintended. That is, student data accumulaced prior to March was no longer 
stored in the system and thus could not be retrieved. In light of these and 
other daily problems (e.g., keyboards freezing, system clock loosing time, 
printers malfunctioning, no boot disc for 2.9 software to use with home 
learning computers), it was not possible to fully implement the project. 
Thus, in spite of the district's ongoing concern and efforts, the district 
and R3S agreed that it would be misleading to report and analyze outcome 
data . 

The Pittston Area School District began implementing the mathematics 
portion of the Jostens curriculum at the beginning of the school year. 
Early on, this district also experienced some difficulties with the computer 
hardware which resulted in students periodically loosing all of their 
computer work. The source of the problem was finally diagnosed in December; 
in January, IBM replaced a defective file server. A second problem facing 
this district concerned scheduling students on the system. As previously 
indicated, the evaluation design was to include an experimental or "high 
use" group receiving one hour of computer time a week and a control or "low 
use" group receiving 15 minutes of computer time a week. However, due to 
the large number of students and limited number of students per class, it 
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was difficult to schedule students for 60 minutes of computer time a week. 
One proposed solution, to rotate students on satellite stations, was 
rejected because district and school administrators did not want students to 
leave their regularly scheduled classes for computer assignments. During 
informal interviews, mathematics teachers reported that the two student 
groups were generally on 15 and 30 minute schedules, respectively. In 
addition, these teachers were* very positive about the mathematics software 
and indicated that students enjoyed working on the computer. Student 
observations supported this finding of high student interest. 

This district also had difficulty implementing the home learning 
portion of n . project. Problems here related to the computer software and 
to scheduling. first, only five of the ten "home" computers could be 
allocated to students as only five of the ten software discs were compatible 
with the school computer system. Because the other five were updated, 
student work could not be transferred to the main system. Second, the 
computer coordinator experienced difficulty scheduling parent- student 
meetings to discuss student and parent involvement in the home component; at 
least three of which were canceled. As a result, only one rotation of frve 
computers was used at home, from September 10, 1991 to December 20, 1992. 

In January, upon installation of the Jostens reading curriculum, 
another major hardware problem occurred; two of the three computers in each 
classroom were not operating properly and could not be used. Again, IBM 
diagnosed the problem, and replaced the defective base band extenders. In 
February, when the system returned to "normal," teachers discovered that 
most students* placements in the Jostens reading curriculum were not 
consistent with embedded curriculum test results and student ability. After 
moving students to appropriate levels, they were then able to work on the 
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reading curriculum for the remainder of the school year (although also not 
on the 15 and 60 minute time schedules). During interviews, reading 
teachers reported that most students had one 15 minute session a week. 
Unlike the mathematics teachers, most reading teachers were not as pleased 
with the content of the Jostens curriculum, particularly teachers of older 
students, as the reading curriculum only goes through level 6,0. Moreover, 
these teachers felt that portions of their lessons (e.g., introducing new 
skills) and some entire lessons (e.g., test reviews) could not be missed for 
computer time. Reading teachers also frequently expressed a need for more 
information and support for implementing the program. 

It should also be noted that during the second half of the school year 
the teaching staff became aware of the district's budget problems. The 
impact of this tenuous financial situation on the project was two-fold; 
first, the project director was given responsibility for working on the 
district budget, which became very time consuming and thus allowed little 
time for project activities; and second, because rumors of cost-cutting 
efforts and layoffs were cause for tension and concern among school staff, 
administrators were reluctant to strictly enforce a new project. 
Program Outcomes 

Specific information on outcomes resulting from the Pittstcn Area 
School District's implementation of the computer project was gained from 
attitude surveys, the Stanford Achievement Test (SAT), and Jostens student 
achievement reports. The sections below describe and discuss analyses of 
this survey and achievement data. 

Survey Data . Attitude surveys were administered to administrators , 
teachers, students, and parents at the end of the school year. These data 
are briefly described below, and where appropriate, are compared to the year 




one survey results. A summary of all quantifiable survey data (i.e., mean 
ratings) is presented in Appendix A. 

The three administrators in the Pittston Area School District who 
completed the survey reported having little experience in using computers. 
In terms of their knowledge of the computer learning system, two of the 
three reported that they knew enough about the Jostens system and that the 
staff received adequate training. Of the 20 potential concerns about the 
new computer learning system listed on the survey form, at least two 
administrators agreed with fourteen (i.e., a rating of 4 or 5 on the 5-point 
scale). The items not of concern related to their own knowledge of 
computers and computer hardware and software, and their need for technical 
assistance. In their comments, they agreed that the major advantage of the 
system was its potential to improve achievement by meeting individual needs. 
One administrator listed a disadvantage; he felt that the system was 
"somewhat disruptive to traditional classroom and scheduling." There were 
no recommendations for changes. 

One administrator did not respond to the remainder of the survey. 
Although the two that did respond agreed that the system documents and 
achievement reports were useful to the staff and to them personally, and 
that implementation and operation of the system did not create problems for 
district administrators, they disagreed on the instructional value of the 
mathematics curriculum (this was the only curriculum area addressed by both 
respondents) . That is, one administrator felt that the computer lesson 
strategies did not support the school's instructional methods, that the 
program was not flexible enough to be aligned with the classroom program, 
and that the computer system's lessons did not adequately prepare students 
for standardized achievement tests. As an explanation., this administrator 
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commented that the system is still experimental. In contrast, the other 
administrator responding to the items noted that the "program is thorough - 
curriculum follows standards established by leaders in above-listed content 
areas. This is what I would expect from Jostens," Perhaps these 
conflicting administrator views on the value of the computer learning system 
were being transmitted to school staff and students. 

Of the twelve teachers who completed the survey, seven reported having 
some computer experience, one reported having very little experience, and 
four reported having no prior computer experience (mean rating of 2.3 on a 
5-point scale). Seven of the 20 concerns about the new computer leai mg 
system listed on the survey were reported by the majority of respondents to 
be strong concerns (i.e., a rating of 4 or 5 on the 5-point scale). Four of 
these concerns related to teachers' own knowledge and preparation about 
computers, about using the software, about their own training and follow-up 
technical assistance, and about using the computer print-outs; most teachers' 
also indicated interest in learning more about the syscem. Two other 
concerns were scheduling of students on the system, and the alignment of the 
Jostens curriculum with the district's curriculum and testing. 

The teachers listed the major advantages of the system to be 
reinforcement of skills , increase of student involvement and motivation, and 
individualization. Only two teachers cited disadvantages: one felt that 
students needed more computer time, the other found scheduling computer time 
to be difficult. Three teachers offered recommendations for improvement: 
one felt that the headsets should be disinfected after each use; another saw 
a need for more computers both in the classrooms and in the computer lab; 
and the third would like to see a better correlation between the district 
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and computer curriculum. It should be noted that many teachers felt that it 
was too soon to comment on implementation, recommendations, and changes. 

The student survey was completed by 316 sixth, seventh, and eighth 
grade students from the high use and low use groups. Overall, student 
responses to the survey's 30 "yes-no" items were positive, although somewhat 
less positive than those of last year. Students indicated that the computer 
is easy to use, that they like computer work better than written 
assignments, and that they can do most of the computer lessons without help 
from anyone else (89 percent or more of the students responded positively to 
these items). Most students (82 percent) also recognized that it is 
important to do well on computer assignments. Fif freight percent of the 
students felt that the computer helped them to learn math better and 
thirty-eight percent felt it helped them to read better. The biggest 
changes from last year were: an increase of 18 percent in students who feel 
that their teacher knows when they make mistakes on their computer 
assignments; and a decrease of 18 percent or higher in students who feel 
that computers make it fun to learn, computers make school subjects more 
interesting, who get bored working on the computer by themselves, who like 
going to the computer, who would like to go to the computer more often, and 
who find computer lessons interesting. It should be noted that, in spite of 
these decreases, which probably reflect more realistic attitudes than the 
year one baseline attitude data, the majority of students responded 
positively to all of these survey items. 

Most parents learned about the project through an initial letter 
describing the project and requesting permission for their child's 
participation and through periodic progress reports. In addition, parents 
whose children were selected to participate in the home learning portion of 
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the project were invited to a student-parent group meeting. Although the 
district did not administer the parent survey, the parents of the five 
children who took computers home completed the home use survey. These 
parents agreed that it was very easy for their children to learn to use the 
computer and that their children used it very often (i.e., all ratings of 5 
on a 5-point scale). The respondents also indicated that their children 
enjoyed working on the computer system at home. In terms of impact, three 
parents felt that their children learned from using the computer system at 
home; two of these parents reported that mathematics skills in particular 
were improved; the remaining parents did not comment on these items. 

Achievement Data , The first type of achievement data analyzed was the 
Jostens student achievement reports. The rationale for looking at these 
data is that if the computer learning system is to have an impact on student 
achievement in general, it must first be demonstrated that students made 
substantial progress in the system, both in terms of lessons completed and 
time spent working on the Jostens curriculum. For this district, analysis 
of time data is particularly important in that it reflects on the validity 
of the experimental design, i.e., the 60 minute experimental condition and 
the 15 minute control condition. 

Performance records were maintained by the system for each student's 
interaction with the curriculum and the level of lesson in which the student 
was engaged. The district provided RBS with students' beginning of year 
placements and last lessons completed for mathematics. As indicated in last 
year's report (Beyer, 1991), most students participating in the project were 
placed on the fourth grade level; the mean placement levels for the Basic 
Skills Inventory ( the curriculum- embedded tes t ) for mathematics , for the 
three grade levels, were from 2.5 to 3.5 grade levels below students' actual 
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grade level and, as the actual grades increased, the initial placements 
relative to them decreased. Although, in the Jostens curriculum, the number 
of lessons per unit and units per grade are not consistent from grade to 
grade, the number of lessons completed was felt to be one of the best 
available measures of student progress in the system. Table 1 presents 
these data for the high use and low use groups, and includes the number of 
students in each group (N), the mean number of lessons completed, the 
standard deviation (SD) , and a t test for independent samples at each grade 
level. This analysis could not be carried out for reading as initial 
reading placements were not provided to RBS, 

Table 1 

Mean Number of Mathematics Lessons Completed for Experimental (High Use) 

and Control (Low Use) Groups 



Grade/ 
Group 


N 


Lessons 
Mean 


Completed 
SD 


t 


6th Grade 










High Use 


52 


66 


47 


3,41* 


Low Use 


54 


.40 


31 




7th Grade 










High Use 


37 


25 


13 


ns 


Low Use 


45 


21 


25 




Sth Grade 










High Use 


48 


52 


56 


ns 


Low Use 


43 


35 


43 





* p < .05 
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As the table indicates, the sixth grade and eighth grade high use 
groups completed more lessons over the course of the year than the 
corresponding low use groups, and this difference was statistically 
significant at the sixth grade level; both seventh grade groups completed 
very few mathematics lessons. Also, the standard deviations are quite large 
indicating a great deal of variability in the number of lessons completed by 
individual students in each group. In interpreting these data i**. should be 
noted that the Jostens Student Achievement Report Manual recommends that 
students complete 12 to 15 lessons over an instructional interval of 4 
weeks. It goes on to indicate that students completing fewer than 10 
lessons during- a 4-week period will be progressing too slowly to make 
meaningful progress. If this guideline was followed, the appropriate time 
periods for the three high use groups in Table 1 would be extremely short - 
6.6 weeks for 6th grade, 2.5 weeks for 7th grade, and 5.2 weeks for 8th 
grade - in contrast to the approximately 28 weeks (i.e., 7 months) of 
reported implementation of the Jostens mathematics curriculum. Thus, the 
system was being implemented by high use students at a level much lower than 
that recommended by Jostens for achieving academic growth. 

Additional analyses were carried out on the mean amount of time 
students spent completing the Jostens reading and mathematics lessons (time 
data were available for reading as well as for mathematics). Unlike the 
lesson data described above, these data were provided by summary reports and 
did not have to be calculated individually for each student. Tables 2 and 3 
present mean time in minutes that students were engaged in mathematics and 
reading lessons (test times are not included). Specified in the tables are 
the number of students in each group (N), the mean number of minutes 



18 O 



1 

I 

i 

1 

1 

i 

1 



Table 2 

Mean Mathematics Time for Experimental (High Use) 
and Control (Low Use) Groups 



Grade/ 
Group 


M 
li 


Mathematics Time 
Mean 


(Minutes ) 
SD 


t 


U LIL V> J. d \J C 










High Use 


59 


557 


164 


7.00* 


Low Use 


62 


356 


151 




7th Grade 










High Use 


43 


302 


148 


3.26* 


Low Use 


50 


205 


138 




8th Grade 










High Use 


54 


370 


181 


3.67* 


Low Use 


50 


255 


137 





p < .05 
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p < .05 



Table 3 

Mean Reading Time for Experimental (High Use) 
and Control (Low Use) Groups 



Grade/ 
Group 


N 


Reading Time 
Mean 


(Minutes ) 
SD 


t 


6th Grade 










High Use 


62 


145 


98 


ns 


Low Use 


' 59 


135 


46 




7th Grade 










High Use 


50 


117 


142 


ns 


Low Use 


43 


79 


110 




8th Grade 










High Use 


50 


113 


154 


ns 


Low Use 


54 


78 


112 
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students spent completing lessons, the standard deviation (SD) , and a t test 
for independent samples at each grade level. 

Examination of the data displayed in Table 2 indicates time spent on 
the Jostens mathematics curriculum, for the high use groups, ranged from a 
little more than nine hours (557 minutes) to five hours. (302 minutes). 
Assuming the mathematics portion of the program was in operation for 
approximately seven months, these data suggest that most high use students 
interacted with the curriculum for less than one hour a month. Although the 
three high use group times were significantly higher than that of the 
controls (low use groups), the means are one and one half times higher and 
not three or four times higher as would be expected (i.e., to reflect the 15 
and 60 minute conditions). Thus, these time data also support the fact that 
the program was not fully implemented. Interestingly, although all three 
comparisons were significant for mean mathematics time, only the sixth grade 
comparison was significant for mean number of mathematics lessons completed 
(see Table 1). As with the lesson data, it should also be noted that the 
large standard deviations reflect a high within group variability. 

Although the reading portion of the Jostens curriculum was operational 
during the second half of the school year, it is clear from the data in 
Table 3 that it was minimally used by students; mean times for the high use 
groups ranged from about two and a half hours (145 minutes) to a little less 
than two hours (113 minutes). These time data for mathematics and reading 
are graphically represented by frequency histograms in Figures 1-6. The 
histograms make immediately clear that most students had six hours or less 
of mathematics computer time, with the exception of the sixth grade high use 
group, and two hours or less of reading computer time. 
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In addition to student progress on the Jostens curriculum, another 
focus of the evaluation is on the extent to which progress on the computer 
system is transferable to standardized achievement tests. Tables 4 and 5 
present analyses of SAT mathematics and reading scores in terms of normal 
curve equivalents (NCEs) by grade and group, for tests administered in the 
spring of 1991 (pretest) and 1992 (posttest). Students' scores are matched 
for the two years so that overall achievement gains can be examined for 
students participating in the program for more than one year. The small 
number of seventh and eighth grade students who did not have data available 
for both years were eliminated from the analyses. Thus, for this second 
year evaluation, grades 7 and 8 have participated in the project for two 
years; for the year three evaluation, grade 7 will have participated for two 
years and grade 8 will have participated for three years. 

NCEs are normalized standard scores, with 50 indicating the national 
average or grade level. An NCE gain of zero from one year to the next would 
signify that students maintained the same relative standing with respect to 
the norm group, which is what would be expected to occur without any special 
program or intervention. As can be seen in Tables 4 and 5, the SAT results 
failed to show any significant NCE gains, from pretest to posttest. Upon 
examining the grade comparisons for the high use groups, seventh gra Je 
mathematics scores showed a statistically significant decrement from the 
expected level of achievement, and the others showed non- significant changes 
ranging from +3 to -1 NCEs. The pattern for the comparison (low us<j) group 
was similar, with seventh grade mathematics scores also showing a 
statistically significant decrement from the expected level of achievement . 
Also, in three out of the four comparisons the size of the gains was smaller 
for the low use groups as compared to the high use groups. In summary, 

27 



1 



i 

1 

g 

1 



Table 4 

Mathematics Achievement Test Scores (SAT) for Experimental 
(High Use) and Control (Low Use) Groups 



Grade/ 
Group 



7th Grade 
High Use 
Low Use 



41 
42 



SAT (Mean NCE) 
1991 1992 



52 
53 



46 
45 



NCE 
Gain 



-6 
-8 



4.63* 
7.84* 



8th Grade 
High Use 
Low Use 



49 
43 



47 
46 



46 
47 



-1 
+ 1 



ns 
ns 



p < .05 



Table 5 

Reading Achievement Test Scores (SAT) for Experimental 
(High Use) and Control (Low Use) Groups 



1 

i 



Grade/ 
Group 



7th Grade 
High Use 
Low Use 



N 



42 
41 



SAT (Mean NCE) 
1991 1992 



44 
46 



45 
44 



NCE 
Gain 



+1 
-2 



ns 
ns 



8th Grade 
High Use 
Low Use 



43 
49 



44 
47 



47 
49 



+ 3 
+ 2 



ns 
ns 
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p < .05 



28 



3b 




s 



i 



I 



1 

■ 



1 
p 

1 

o 

ERIC 



there is no evidence of accelerated achievement levels on the SAT resulting 
from the computer learning system program. In addition, these data show 
that both seventh grade groups began the program with above average 
mathematics achievement levels and the following year had a significant 
decrease to below grade level performance. 

Conclusions and Implications 
During year two of the CAI/Cooperative Learning Project, the two 
participating school districts experienced many difficulties with t the 
Jostens Learning Corporations* computer-based learning system. These 
problems involved both the computer software and hardware. As a result of 
these and other district problems, one district, the Hatboro-Horsham School 
District, reached only a minimal level of implementation and was not ready 
to document the impact of the system; the second district, the Pitt; ton Area 
School District, had a higher level of implementation, yet analyses of the 
computer learning system reports suggest these students spent little time 
working on the system - student use was far below the level deemed to be the 
minimal requirement as recommended by the Jostens Manual (although the 
discrepancy between staff reports of student use and Jostens' documentation 
is puzzling) . Because the program was not fully implemented, the drawing of 
any specific conclusions about the impact of the computer learning system is 
not possible. 

This second year evaluation has several implications for the project in 
year three: 

• Although computer-assisted instructional programs such as these 
commonly experience setbacks and delays, the amount of difficulty 
experienced by these two districts is exceptional. For the program 
to succeed, a district priority must be to have the project fully 
operational and successful. Thus, Jostens Learning Corporation's 
full cooperation and support must be engaged until the networked 
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systems are functioning properly; their ongoing assistance with 
regard to curriculum planning and implementation is also essential. 

• The district administrations must communicate this priority and 
positive attitude toward the project to the school staff and 
students. Clear program goals and expectations must also be 
communicated. To succeed, the project must have leadership. 

• In the Pittston middle school, the computer coordinator must 
increase his involvement with the program and closely moaitor 
teachers ' use of the computer learning system. For example , both 
the year one and year two evaluations found that teachers have 
positive attitudes toward the computer learning system but have 
specific needs for additional information and support. Perhaps 
regularly scheduled meetings with the computer coordinator (e.g., to 
monitor and review reports, answer questions, monitor 
individualization) would be a successful strategy to provide for 
both project leadership and staff support. 

• The Pittston Area School District needs to monitor and support the 
implementation of the evaluation design which it selected for the 
study, i.e., a 15 minute control group and a 60 minute experimental 
group. The meaningfulness of the study rests on the validity of the 
design - a control group minimizes threats to internal and external 
validity. The computer system must be used for reading and 
mathematics instruction at or above the level deemed to be the 
minimal requirement recommended by the computer learning system 
vendor. Also, because these data are different than those expected 
by project staff, the accuracy of the Jostens computer performance 
reporting system must be examined, 

• The selection and composition of the program and comparison groups 
should be closely monitored to maintain consistency over the three 
year period, particularly with regard to group size and achievement 
level. For example, the sixth grade groups at the Pittston middle 
school had mean mathematics scores on the standardized test of 56 
and 58 NCEs, which do not reflect "at-risk" performance. 

• The cooperative learning component of the program needs to be 
adequately addressed. Although each district has isolated examples 
of its successful use, there is no plan for implementing and 
monitoring cooperative iearning. That is, teachers are unclear as 
to how cooperative learning is defined, and how it relates to 
students' involvement with the computer learning system. Likewise, 
parent involvement in cooperative learning at home also needs to be 
addressed . 

This evalu tion report obviously recognizes that much of the difficulty 
experienced by the two districts may be beyond their control, that is, a 
result of Jostens* premature marketing both of the software and the 
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networked system. It also recognizes that both districts exerted much time 
and effort trouble shooting basic problems which should not have occurred 
and which "snowballed" into larger, unanticipated outcomes. 
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Pittston School District (N=3) 
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Item Mean 

A. 4 . To what extent are you experienced in using 3.7 

computers? 

B. l. I aon't know enough about the computer learning 2.3 

system. 

B.2. I am concerned about the attitude of students toward 4.3 
the computer system. 

B.3. I know of other approaches that might work better. ' 3.3 

B.4. I have very limited knowledge about computers. 3.0 

B.5. I am concerned about the effects of the system on 5.0 
students . 

B.6. I am concerned about the changes this system will 4.7 
cause teachers to make in their classrooms. 

B.7. I am concerned about how much time and paperwork 3.3 
the computer system takes. 

B.8. I am concerned about how the scheduling of students 4.7 
on the system works out. 

B.9. I am concerned about the time and energy commitments 4.0 
the computer system requires. 

B.10. I am concerned about the knowing how to use the 3.0 
hardware . 

B.ll. I am concerned about knowing how to use the software. 3.7 

B.12. I am concerned about the curriculum content of the 4.7 
computer learning system. 

B.13. I am concerned about the instructional approach used 4.7 
by the computer learning system. 

B.14. I am concerned about the training I receive on the 3.7 
computer learning system. 

B.15. I am concerned about the follow-up technical 3.0 
assistance I receive on the computer learning system. 

B.16. I am concerned about how well the computer system's 4.7 
curriculum aligns with our district curriculum and 
testing. 
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B.17. I am concerned about the use of computer printouts 4.3 
from the system. 

B.18. I would like to know how this system is better than 2.7 
what we had before. 

B.19. I am not interested in learning more about the system. 1.3 

B.20. I am concerned about the value of instruction 4.7 
students receive on the system. 

D.l. How many hours of training did you receive to prepare 4.0 
you, personally, for your role in use of the computer 
system? 

D.2. To what extent do you feel the training adequately 4*0 
prepared you personally, for your role in use of the 
computer system? 

D.4. To what extent do you feel the training adequately 4.5 
prepared your staff for use of the computer system? 

D.5. To what extent has the inservice training your staff 
received adequately covered the following areas? 

Reading 5.0 

Hath 5.0 

Writing 5.0 

Science 1«0 

Higher Order Thinking Skills 3.0 

Use of Student results reports 5.0 

Integration of computer curriculum with classroom 5.0 
program 

Hands-on experience with the computer sysem 4.0 

Scheduling of students 4.0 

D.7. To what extent do you feel the follow-up technical 3.0 
assistance provided after training was adequate to 
meet the needs of you and your staff? 

D.8. To what extent has there been a need for follow-up 
technical assistance in the following areas? 

Reading 3.0 
Math 3.0 

Ur ih'no 3.0 

Science 

Higher Order Thinking Skills 3.0 

Use of Student results reports 3.0 

Integration of computer curriculum with classroom 2.0 
program 

Hands-on experience with the computer system 2.0 

Scheduling of students 2.0 



2 
2 
2 
2 
2 
2 
2 

2 
2 



2 
2 
2 
3 
1 
1 
1 

1 
1 
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D. 9, To what extent have the following documents met your 

staffs needs? 

Curriculum Guides 
Teacher Guides 
Teacher Handbooks 

E. 2. To what extent are the reports clearly organized and 

easy to use7 

E.3. To what extent have the reports enabled your staff to: 

Diagnose student needs 
Individualize instruction 
Group students by ability 
Inform parents of student progress 

E.4. To what extent have the reports been useful to you 
as an administrator? 

E.5. To what extent has the computer system resulted in 
reallocation of time in the following areas? 

Reading 
Math 
Writing 
Science 

E.6. To what extent has the computer system resulted in . 
increased time-on-task for all students in the 
following subject areas? 

Reading 
Math 
Writing 
Science 

E.7. To what extent do students enjoy working on their 
computer lessons? 

E.8. To what extent do you observe students working on 
lessons at the computer? 

E.9. To what extent has the computer system hardware been 
reliable and dependable? 

E.10. To what extent has implementation and operation of 
the computer system created administrative problems 
for you? 



4.0 
4.0 
4.0 

4.5 



4.5 
4.0 
2.5 
4.5 

4.5 



1.0 
1.0 
1.0 
1.0 



5.0 
5.0 
5.0 
1.0 

5.0 



2.5 



4.0 



1.0 



1 
1 
1 
1 



2 
1 
2 
2 



2 
1 

2 
2 
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F.l To what extent is the computer curriculum sufficiently 
comprehensive to provide adequately for the learning 
needs of students in each of the following subjects? 

Reading 
Math 
Writing 
Science 

F.2. To what extent are the learning strategies and models 
presented in the computer lessons supportive of the 
instructional methods used in the school in each of 
the following subjects? 

Reading 
Math 
Writing 
Science 

F.3. To what extent is there sufficient flexibility in the 
sequencing of the units that the computer program can 
be aligned with the school's classroom programs in 
each of the following subjects? 

Reading 
Math 
Writing 
Science 

F.4. To what extent do computer system lessons adequately 
prepare students for standardized achievement tests7 

Reading 
Math 
Writing 
Science 

F. 5. To what extent does the computer system curriculum 

meet your expectations ? 

Reading 
Math 
Writing 
Science 

G. l. To what extent is there increased student motivation 

and interest in your school as a result of students 
participating in the computer learning system? 



l 

i 



5.0 
4.5 
5.0 



4.0 
3.0 
4.0 



4.0 
2.5 
4.0 



5.0 
3.5 
5.0 



5.0 
5.0 
5.0 



3.5 



1 

2 
3 



2 
1 
2 
3 



2 
1 
2 
3 



2 
1 
2 
3 
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G.2. To what extent has the computer program led to higher 
student achievement? 



Reading 



4.0 2 



Math 4.0 2 

Writing 
Science 



4.0 2 
3 



G.3. To what extent has the computer system promoted 4.0 
student class participation? 

G.4. To what extent has the computer system promoted 4.0 
greater parent interest and involvement in the 
school? 

G.5. To what extent has the computer system enhanced 3.0 
teacher effectiveness? 

G.6. To what extent has the computer system enhanced 4.0 
teacher communication and cooperation? 
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Teacher Survey 
Pittston Area Middle School (N=12) 
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Item 

A. 4 . To what extent are you experienced in using 

computers? 

B. l. I don't know enough about the computer learning 

system. 

B.2. I am concerned about the attitude of students toward 
the computer system. 

B.3. I know of other approaches that might work better. 

B.4. I have very limited knowledge about computers. 

B.5. I am concerned about the effects of the system on 
students . 

B.6. I am concerned about the changes this system will 
cause teachers to make in their classrooms. 

B.7. I am concerned about how much time and paperwork 
the computer system takes. 

B.8. I am concerned about how the scheduling of students 
on the system works out. 

B.9. I am concerned about the time and energy commitments 
the computer system requires. 

B.10. I am concerned about the knowing how to use the 
hardware . 

B.ll. I am concerned about knowing how to use the software. 

B.12. I am concerned about the curriculum content of the 
compute r learning system. 

B.13. I am concerned about the instructional approach used 
by the computer learning system. 

B.14- I am concerned about the training I receive on the 
computer learning system. 

B.15. I am concerned about the follow-up technical 

assistance I receive on the computer learning system. 

B.16. I am concerned about how well the computer system* s 
curriculum aligns with our district curriculum and 
testing. 



Mean 
2.3 

3.5 

2.6 

1.8 
3.6 
3.2 

3.0 

2.6 

3.7 

2.9 

3.5 

3.7 
2.9 

2.8 

3.3 

3.5 

3.3 
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B.17. I am concerned about the use of computer printouts 3.3 
from the system. 

B.18. I would like to know how this system is better than 2.5 
what we had before. 

B.19. I am not interested in learning more about the system. 1.3 

B.20. I am concerned about the value of instruction 2.9 
students receive on the system. 
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Student Survey 
Pittston Area Middle School (N=316) 



Item 

"1. Do you like school? 

2. Is the computer easy to use? 

3. Is working on the computer fun? 

4. Do computers make it fun to learn? 

5. Do you learn a. lot on the computer? 

6. .Do computers make school subjects more 

interesting? 

7. Do you get bored working on the computer by 
yourself? 

8. Do you need much help when working on the 
computer? 

9. Does your computer give you help when you 
need it? 

10. Does the computer help you correct your mistakes? 

11. Do you have to hurry when you work on the 
compute r? 

12. Do you like computer work better than written 
ass ignments ? 

13. Is it important to do well on your computer 
as signments ? 

14 . Does working on the computer help you do better 
in school? 

15. Does your teacher know whether you make mistakes 
on your computer assignments? 

16. Do you get good grades when you work hard in 
school? 

17. Do you like going to the computer? 

18. Would you like to go to the computer more often? 

19. Have you worked on a computer in school before 
this year? 



Percent Responding 
Yes No NR 

50 47 3 

93 

72 

69 

60 

61 



50 

13 

76 

79 
36 

90 

82 

52 

71 

85 

71 
67 
85 



7 
28 
31 
40 
38 

49 

82 

23 

20 
64 



17 

47 

25 

15 

27 
33 
15 
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20. 


Do you have a computer at heme? 


36 


64 




21. 


Do you like using the computer at school? 


73 


26 


1 


22. 


Has the computer helped you to learn math better? 


58 


40 


2 


23. 


Has the computer helped you to read better? 


38 


60 


2 


24, 


Has the computer helped you to write better? 


15 


83 


2 


25. 


Has the computer helped you to understand 
science better? 


16 


80 


4 


26. 


Can you do most of the computer lessons without 
help from anyone else? 


89 


11 




27. 


Are the computer lessons interesting? 


59 


39 


2 


28. 


Do your computer lessons help you do work in 
the classroom better? 


45 


52 


3 


29. 


Is your time with the computer the best part 
of your day? 


27 


73 




30. 


Do your parents think you are learning from the 


57 


36 





computer? 
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Computer Home Use Survey 
Pittston Area Middle School (N=5) 



Item 

B.2. How difficult was it for your child to learn how 5.0 
to use the computer system? 

B.3. Hew often did your child us the computer system 5.0 
at home? 

B.7. How frequently did you work with your child during 2.8 
use of the computer system? 

B.8. To what extent did your child enjoy working on the A. 8 

computer system? 

B.9. To what extent did your child learn from using the 4.7 
computer system at home? 

B.10. To what extent did your child's math skills improve 4.5 
from working on the computer system at home? 

B.ll. To what extent did your child's reading skills improve 0 
from working on the computer system at home? 

B.12. To what extent did your child's writing skills improve 0 
from working on the computer system at home? 

B.13. To what extent did your child's understanding of 0 
science improve from working on the computer system 
at home? 



Mean NR 



53 

A3 



